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The views expressed in this paper are those of the authors and do not 
necessarily reflect the position of the Science and Education Administration 
or the United States Department of Agriculture. 


PREFACE 

This paper was prepared at the pantaee of Dr. Mark T. Buchanan, 
Director-at-Large of the Western Association of Agricultural Experiment 
eration Directors. It was presented as one of five papers in a program 
on "Federal Funding Philosophies, Policies, and Procedures: Impact on 
Research in the Food and Agricultural Sciences" at the annual meeting of 
the American Association for the Advancement of Science (AAAS), San 
Francisco, California, January 7, 1980. 

We have received very helpful comments from Dr. Buchanan and the 
other participants in the AAAS program, from our colleagues in Joint 
Planning and Evaluation, and from a number of other people. B.R. Eddleman 
and Vernon W. Ruttan provided especially thought-provoking comments and 
Suggestions that have been very useful in revising the paper. The ideas 
expressed in the paper are those of the authors and do not necessarily 
represent the official position of the Science and Education Administration 


and the U.S. Department of Agriculture. 


EVALUATION AS AN AID TO DECISIONMAKING IN THE 
FOOD AND AGRICULTURAL SCIENCES 


EXECUTIVE SUMMARY 


A new decisionmaking process is beginning to emerge in food and 
agricultural science and education. The evaluation of economic, social, 
and environmental impacts is an integral part of that process. A series 
of changes are taking place that represent a significant redefinition of 
the decisionmaking process and that suggests future changes in evaluation 
as a part of that process. These changes include new institutions that 
have resulted from Title XIV of the Food and Agriculture Act of 1977--the 
Committee on Food and Renewable Resources, the Joint. Council on Food and 
Agricultural Sciences, and the National Agricultural Research and Extension 
Users Advisory Board. 

Longstanding understandings among the Federal and State partners in 
the funding and provision of agricultural research are being questioned 
and renegotiated. The composition of that partnership is expanding to 
include a broader set of decisionmakers-——those who are stakeholders in the 
outcomes of agricultural science and education. They include providers of 
funds, providers and users of agricultural science and education services, 
and people and institutions impacted by the food and agricultural sciences. 

The emerging framework eae decisionmaking will mean that decisions 
about agricultural science must be informed both about the quality of 
agricultural research, extension, and teaching programs, and about their 


economic, social, and environmental impacts. 
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EVALUATION AS AN AID TO DECISIONMAKING IN THE FOOD 
AND AGRICULTURAL SCIENCES 


James Nielson and John M. Brazzel* 


This paper will first, describe recent changes in the decisionmaking 
process in the food and agricultural sciences and key relationships among 
participants in that process; then, examine the current status of the 
evaluation of focd and agricultural research; and finally, suggest future 


directions in evaluation as a part of the decisionmaking process. 


The Decisionmaking Process 


Recent Changes--Title XIV, Food 
and Agriculture Act of 1977 


The institutional framework within which decisions are made about food 
and agricultural research is changing as a result of innovations called for in 
Title XIV of the Food and Agriculture Act of 1977. These innovations include: 

1. Committee on Food and Renewable Resources 

Congress directed the establishment of the Committee on Food on 
Renewable Resources of the Federal Coordinating Council for 
Science, Engineering, and Technology. The Committee's function 
is to bring about better planning and coordination of research 
in food, renewable resources, and nutrition among agencies of 
the Federal government. It includes representatives from 12 
Federal agencies. 

* James Nielson is Deputy Director for Joint Planning and Evaluation, 
Science and Education Administration, U.S. Department of Agriculture, Executive 
Director of the Joint Council on Food and Agricultural Sciences and the 
National Agricultural Research and Extension Users Advisory Board, and Executive 
Secretary of the Committee on Food and Renewable Resources of the Federal 
Coordinating Council for Science, Engineering, and Technology. John M. Brazzel 


is Chief of the Evaluation and Impact Staff, Joint Planning and Evaluation, 
Science and Education Administration, U.S. Department of Agriculture. 
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Joint Council on Food and Agricultural Sciences 
Congress believed that there was unnecessary duplication in agricultural 
research so it directed the establishment of the Joint Council on Food 
and Agricultural Sciences to foster improved planning and coordination. 
This occurred in spite of testimony from the U.S. Department of Agriculture 
(USDA) and university representatives that a new congressionally-mandated 
body was not needed. 

Congress directed that the Joint Council foster regional and national 
planning and coordination of agricultural extension and teaching activities 
as well as research, and that non-land-grant universities and represen- 
tatives of private industry participate along with the more traditional 
land-grant and USDA partners. The Joint Council also was given a 
charge to "analyze and evaluate the economic, environmental, and 
social impacts of agricultural research, extension, and eeschine 
programs...and determine high priority research areas..." 

Congress hoped that unified plans with delineation of responsi- 
bilities developed and agreed upon through efforts of the Joint Council 
would help it in dealing with what appeared to be overlapping and 
competing requests cor Federal funds from various components of 
the agricultural science and education system. 

The Council is alive, well and active. Whether the hopes of 
Congress will be Perit ed through it remains to be seen, but it will 
likely lead to some improvement in planning, coordination and evaluation. 
National Agricultural Research and Extension Users Advisory Board 
Both the legislative and executive branches of Federal government have 
been motivated to obtain greater citizen participation in public 


decisionmaking on policies and programs in general, and on agricultural 





& 
science and education programs in particular. (State and local 
governments, of course, have similar objectives.) One of the 
mechanisms to obtain citizen input is the National Agricultural Re- 
search and Extension Users Advisory Board. The Board prepares inde- 
pendent advisory opinions about program priorities, Federal budgets, 
division of responsibilities, and effectiveness of agricultural research 
and extension programs for use by the Congress, the President, the 
Secretary of Agriculture and by all of the partners in the system. 
It is composed of 21 members who represent users of and those impacted 
by agricultural research and extension activities. Local, State and 
Federal agencies who are users of research and extension services are 
mot represented on the Board. 

For 21 citizens with great diversity of interests and philosophies— 
all of whom have full-time jobs--to arrive at a consensus on national 
research and extension priorities is a major assignment. While they 
still have quite a way to go, their October 1979 report amazed almost 
everyone in its relevance and degree of consensus. 

The creation of the Committee on Food and Renewable Resources, Joint Council, 
and Users Advisory Board provides for the inclusion of a new, much broader set 
of participants in the decisionmaking process of the food and agricultural 
saReeoet These actors—~or "stakeholders" in the outcomes of research, extension, 
and teaching in the food and agricultural sciences—-now include Federal, State 
and private providers of funds; Federal, State, university, and private providers © 
of agricultural science services; users of agricultural science services; and 


people and institutions impacted by the food and agriculture sciences. 
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Title XIV goes beyond the definition of a pron ea tetas participation in 
the decisionmaking process. It also calls rae the integration of extension and 
teaching with research in the decisionmaking process, and calls for the con- 
sideration of economic, social, and environmental impacts of agricultural science 


programs in decisionmaking. 


Contracts Among Participants in the Decisionmaking Process 


The institutional framework for decisionmaking in the food and agricultural 





sciences encompasses a series of implicit and explicit understandings, agreements 


or contracts among the participants. These wnderstandings include the concepts 





of accountability, hence eralesrare partnership, and what Richard Atkinson (2) 
and Philip Handler (5) have termed the social contract between science and society. 
The increased emphasis on accountability for the use of public funds is one 
of the most significant developments of the 1970's. It involves either an implicit 
or explicit contract between providers of public funds for food and agriculture 

science (Federal and State) and recipients of those funds who produce agricultural 
research, extension, and teaching services. Funds are provided through Federal 
and State budgeting processes, which now require more detailed documentation of 
the needs for and uses of funds and anticipated impacts of programs. 

The Federal-State partnership in the provision of agricultural research, 
extension, and teaching involves a long-term, cooperative relationship between 
the U.S. Department of Agriculture and the State land-grant universities that 
began in the late 1800's and early 1900's with the Morrill, Hatch, and Smith- 
Lever Acts. The partnership grew out of the conviction that the joint and 
cooperative provision of agriculture, research, extension, and teaching by the 
Federal and State governments could best provide for the needs of farmers, con- 
Sumers, and society. 

An implicit understanding or social contract has developed over time be- 


tween the science community and society as a part of the increased public 
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involvement in science. This contract acknowledges that the public support of 


science, along with a strong degree of independence of science institutions, are 


in the social interest. The elements of the social contract between science and 


society and its operation are described by Higgins (6, p.5): 


If science is to be a personalized cultural pursuit then its level of 


funding will soon drop.to the levels of other cultural ACCIVGE LES cnet 


on the other hand, science is to be supported as an instrument of social 


change and funded accordingly, then it must respond to societal need, and 


society will insist on the deliberate direction of scientific activity 


towards areas of greatest social need or potential. 


Contract Challenges and Renegotiations 


These understandings are now being challenged by a general concerm and doubt 


among the participants in the decisionmaking process in the food and agricultural 


“sciences. These doubts arise out of: 


1. 


Dwindling rates of increases in Federal budgets for agricultural 
Tesearch and erie especially as related both to apparent needs 
and to the rate of inflation. 

Concern among some State partners as to whether the Federal government 
is as committed as it once was to the Federal-State partnership. These 
doubts appear to stem from reaction to reorganization of the science 
and education erences of the USDA; representatives of the Federal 
government, especially the Office of Management and Budget (OMB), 
making unilateral decisions on the portion of the research agenda that 
will be left to the States; the recent interest in competitive grants, 
as opposed to formla-funding, as a mechanism for providing Federal 
funds for agricultural science and education; and actions by both 

the Legislative and Executive Branches of the Federal. government to 


include nontraditional participants in the decisionmaking process. 
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3. Increased eases on the use of public fumds provided for science. 
4. Changing social goals in the direction of social equity, conservation 
of natural resources, environmental quality, and quality of life and 
away from production, efficiency, and income. 


5. Recognition of what are now perceived as undesirable social and environ- 





mental impacts of some technological advances in agriculture. 
6. A growing public skepticism about the responsiveness of the general 
science community, including the food and agricultural sciences, to 


social needs. 








7. The potentially inherent conflict between views of science as value- 
neutral activities, autonomous and independent of the social systen, 
and the concepts of public and social accowntability. 

These concerns have led to a process of questioning and renegotiating the 
existing understandings and contracts among participants in the decisionmaking 
process in the food and agricultural sciences, as well as an expansion in the | 
scope of those included as participants. The formation of the Committee on | 
Food and Renewable Resources, Joint Council, and Users Advisory Board are examples 
of that response. 

Other institutional mechanisms have appeared in recent years. They include 
public interest groups, such as the Agribusiness Accountability Project and 
Center for Science in the Public Interest; Federal and State public partici-~ 
pation, freedom of information, and consumer affairs laws and regulations; 
public and privately-sponsored workshops, conferences, and dialogues about 
research and technology in agriculture; and an increased interest and involvement 


of Federal and State courts in the conduct of agricultural research. 
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These and additional concerns about the scarcity of public information 
describing the status of the food and agricultural sciences have resulted in 
requirements for enhanced collection of management information, external peer 


teview of programs, and program impact measurement and evaluation. 


‘The Evaluation Process 

The evaluation process in the food and agricultural sciences includes (1) 
the assessment of the technical or scientific quality of programs and (2) the 
measurement and evaluation of the technological, economic, social, and environ- 
mental impacts of the programs. 

The first category of evaluation includes program and external peer 
reviews which can be categorized as process or management evaluation; this 
evaluation category has been described and analyzed in an excellent paper by 
Vernon Ruttan (9). 

In this paper we are focusing on the second category of evaluation -- 
impact evaluation. This section includes a brief description of the major 
elements of an impact evaluation process and an assessment of the status of 
evaluation in the food and agricultural sciences. ee changes that can be 
anticipated in evaluation as part of the decisionmaking process are examined 


later. 


Characteristics of Impact Evaluation 


Good impact evaluation studies* address questions related to the outputs 


and inputs of science programs, such as: 


*Criteria for judging the quality of impact evaluations are suggested in a 
report by the United States General Accounting Office (12, p. 2) to include 
relevance, significance, validity, reliability, objectivity, and timeliness. 

The GAO report provides a useful framework and checklist for determining whether 
these criteria are met by specific impact evaluation studies. . 
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1. Were the objectives of the research program accomplished? 

2. What were the technological, economics social and environmental 

impacts of the program? 

3. Who benefits from the programs, how, and how much? 

4. Who is worse off because of the programs, how and how much? 

>. What were the cost of the programs in terms of investments 

and other opportunities foregone? 

6. Who bears the costs? 

7. How do the benefits compare with the costs? 

Evaluation is an important source of program information that can be useful 
in the decisionmaking process. Uses of impact evaluation results include program ° 
planning, coordination, establishment of priorities, redirection of program 
resources, budget development, and more informed and more timely involvement in 
the decisionmaking process by participants. 

Participants or stakeholders in the decisionmaking process in the food and 
agricultural sciences are also Sraenete ors in the quality, objectivity, and 
results of the evaluation process. These participants include Federal, State, 
and private providers of funds; Federal, State, university, and private Sceroe 
performers of agricultural science; users of research, extension, and teaching 
services; and people and institutions impacted by the food and agricultural 
sciences. | 

An important, second order but highly pragmatic use of evaluation results is 
the documentation of needs for food and agricultural research, extension, and 
teaching budgets. This is a sensitive and delicate issue for the evaluation 
process and must be treated very seriously. Rests from high-quality, objective 


evaluation studies can help in establishing budget needs because the data are 


believable and accepted. Inappropriate uses of evaluation results or the conduct 
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of self-serving analyses under the guise of the evaluation process can destroy 
the credibility and usability of evaluation results as a part of the decision- 
making process. 

In order for evaluation results to be most useful to participants in the 
decisionmaking process, they must be designed and conducted in such a manner 
as to have credibility both with those whose programs are being evaluated and 
those who may use the evaluation results. This implies close adherence to 
scientific method in conceptualizing and conducting the studies. It also re- 
quires the use of evaluation personnel whose main motivation is the professional 


quality and usefulness of the evaluation product. 


Status of Evaluation in Food and Agricultural Science 


Evaluation of the impacts of agricultural science programs within the 
‘food and agricultural science community has been conducted primarily by a few 
agricultural economists and rural sociologists in land-grant universities and 
several nonland-grant universities, and by agricultural economists in the 
USDA. Levels of resources committed to impact evaluations have been quite low. 

For several years during the mid-1970's, each of the four regional asso- 
ciations of State agricultural experiment station directors sponsored research 
evaluation activities by one or two analysts in each region. USDA assisted 
these programs financially ana otherwise. In the West, the regional analyst 
was supported by the State extension directors also, and evaluated both extension 
and research programs. 

The Science and Education Administration (SEA) in USDA has recently formed 
a small, multidisciplinary Evaluation and Impact Staff (EIS) with the respon- 
SLULuLLy, elo part, to conduct impact evaluations of agricultural research, 


extension, and teaching programs. An annual budget has not been allocated 
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at this time for impact evaluation studies in SEA. Funds for this purpose are 
iimited and are negotiated on an ad hoc basis as needs for evaluation studies 
_become apparent. 

New efforts are underway to foster and encourage the development of a 
healthy evaluation process in the agricultural sciences. The Joint Council will 
encourage the evaluation of program impacts, and potentially the formation of 
regional networks of people concerned with the evaluation process, as a part of 
its efforts to foster planning and coordination at both the regional and national 
levels. The Users Advisory Board also will encourage program evaluation. 

The State agricultural experiment stations have formed a new interregional 
research project on Planning, Evaluation, Analysis, and Coordination of Research 
(IR-6). B. R. Eddleman is Director of the project. One of the objectives of 
IR-6 is "to foster research to evaluate the impacts—economic, social, and 
environmental—-of agricultural research programs." 

Ey alie i onieernadotoet es and findings for the agricultural sciences have 
been examined in a series of conferences in 1969, 1975, 1976, and 1978. The 
conference proceedings for the first three conferences are reported in Fishel 
(4), Arndt, Dalrymple, and Ruttan (1), and Back (3). A further conference in 
1980 is being organized by Walter Fishel. The expanding scope and participation 
in evaluation activities in the agricultural sciences are.indicated by a com- 
parison of the proceedings and participants. in these conferences. The conferences 
have involved an increasingly wider group of people within the agricultural 
science community and-some from outside the ee The conference papers 
address studies concerned with technology assessment and domestic and inter- 


national programs. A few studies examine both research and extension programs. 
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The status of evaluation methodology and results in agricultural research 
is reviewed in reports by Higgins (6), Norton and Davis (8), Schuh and Tollini 
(10), and Shumway (11). Significant accomplishments have been made in the 
development and application of methodologies to estimate the impacts of 
agricultural research on agricultural productivity and national income. 
Procedures have been developed for ex ante and ex post evaluation studies. Early 
estimates of the returns to aggregate production-oriented agricultural research 
have been refined. Estimates of the returns to both research and extension are 
now available. Disaggregated estimates for categories of commodities and for 
regions are also available. Methodologies for estimating the returns to 
agricultural research have increased in number and improved in quality. 

Most evaluations of agricultural research have focused on impacts on 
agricultural productivity. Some of the reasons for this focus appear to be 
(1) most of the studies have been conducted by economists, and measurement of 
productivity has been central to the mainstream of economics; (2) throughout 
the 1950s and 1960s, productivity and efficiency were among the major goals of 
our society; (3) evaluation of productivity was facilitated by available data 
bases; and (4) evaluation of research and extension programs, not directly 
related to production, involve measurement and other methodological problems. 

Relatively few studies are available that measure and assess broader 
economic, social, and environmental impacts of agricultural science programs. 
There are Ee era including those in the areas of public policy analysis and 
in studies of the impacts of chemical pesticides and non-point pollution and 
water quality. Madden (7) reviews a. number of studies that have examined 
aspects of the broader economic and social considerations in the area of 


agriculture mechanization. In general, however, little attention has been 


ee 
given to evaluating the broader economic, social, environmental and tech- 
nological impacts of. food and agricultural science. 

The evaluation process in food and agricultural science generally can be 
characterized as operating from a relatively narrow base in view of its: 

1. Emphasis on production impacts, as opposed to broader economic 

and social impacts of programs. 

2. General focus on agricultural research activities, rather than 
the total research, technology development, transfer, and 
adoption process involved in agricultural research, extension, 
and teaching programs. 

3. General isolation of evaluation activities from the planning, 
resource allocation, and budgeting processes in the food and 
agricultural sciences. 

4. Isolation from and a lack of awareness of the impact evaluation 

methodologies applied outside of agriculture, the development 
of Ph.D. programs in evaluation research at a number of — 
universities, and the rapidly-growing evaluation-research 
community throughout the United States. 

While there pevenbeen a number of good, objective evaluations of agricultural 
research and extension, overall cheteredipittes of agricultural science evaluation 
is questioned by Congress, the Office of Science and Technology Policy (OSTP), OMB 
and some of the public impacted by the programs. Perhaps because of the low-level 
commitment of resources to the evaluation process in the food and agriculture | 
sciences, its narrow focus, and its general isolation from evaluation methodo- 
logies used outside of agriculture, there is a general and pervasive skepticism 
about the objectivity of current evaluation efforts in the food and 


agricultural sciences. 
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Future Directions in Evaluation as 


Part vor ithe Decisionmaking Process 


Evaluation is at the center of the dialogue that is taking place about the 


decisionmaking process in agricultural science and education. Changes that are 


occurring today and that have been discussed in this paper suggest future patterns 


in impact evaluation that will take place as a part of the evolving changes in the 


overall 


Ls 


decisionmaking process. They include: 

Expansion of the traditional partnership in the agricultural science 
and education system, and increased public involvement in agricultural 
Science decisionmaking and evaluation at Be toe. State and local 
levels. 

Increased numbers of specifically-defined evaluation studies required 
by Congress, the Office of the President (OMB and OSTP), State 
governments, the Joint Council and the Users Advisory Board. 

Expanded and earmarked Federal and State Budgets used for evaluation 
research at national and State levels. 

Development of an integrated, trained evaluation-research community 
and network among the local, State and Federal partners in the system. 
Decisionmaking about the evaluation of agricultural research at 
national, regional, and State levels in the context of the overall 
agricultural Peer cre extension, and teaching process involved in 


the food and agricultural sciences. 


Concluding Comments 


A new decisionmaking process is beginning to emerge in the food and 


agricultural sciences. The evaluation of economic, social, and environmental 


impacts 


is an integral part of that process. We have examined a series of 


14 


changes that are taking place that represent a significant redefinition of the 
decisionmaking process and have suggested future directions in evaluation A 
part of that process. 

Longstanding implicit and explict contracts among the Federal and State 
erred in the funding and provision of agricultural research are being 
questioned and perevoeiaceds The Seabcettet of that partnership is expanding 
to include a spgane: set of decisionmakers—those who are stakeholders in the 
outcomes of agricultural science and education. They include providers of 
funds, providers and users of agicultural science services, and people and 
institutions impacted by food and agricultural science. 

The emerging framework for decisionmaking will mean that decisions about 
agricultural science must be ee both about the quality of agricultural 
research, extension, and teaching programs, and about their economic, social, 
and environmental impacts. 

A well-trained and supported group of evaluation researchers will develop 
in the agricultural science commmity during the 1980s, along with a growing | 
body of good, objective impact evaluation studies. 

Special attention will need to be given to insure that these studies are 

designed and presented in a form that facilitates their use in the 


decisionmaking process. 
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